Intruding object detecting method and intruding object monitoring apparatus employing the method

ABSTRACT

An intruding object detecting method and an intruding object monitoring apparatus for detecting a target object intruding into an image pickup region while reducing the error detection of moving objects other than the target object. A difference in pixel value between an input image signal and each of different image signals in a predetermined number of frames greater than one is calculated for each pixel to thereby obtain differential images in the predetermined number of frames. The differential images in the predetermined number of frames are synthesized in predetermined proportions to thereby generate a synthesized differential image. The synthesized differential image is binarized on the basis of a predetermined threshold value to thereby generate a binarized image. A binarized object in the binarized image is detected as an object intruding into a monitoring visual field.

CROSS-REFERENCE TO RELATED APPLICATIONS

This invention relates to the following U.S. Patent Applications.

Patent application Ser. No. 09/078,521, filed on May 14, 1998, in thenames of Wataru Ito, Hirotada Ueda, Toshimichi Okada and Miyuki Endo andentitled “METHOD FOR TRACKING ENTERING OBJECT AND APPARATUS FOR TRACKINGAND MONITORING OBJECT”;

Patent application Ser. No. 09/392,622, filed on Sep. 9, 1999, in thenames of Wataru Ito, Hiromasa Yamada and Hirotada Ueda and entitled“METHOD OF UPDATING REFERENCE BACKGROUND IMAGE, METHOD OF DETECTINGENTERING OBJECTS AND SYSTEM FOR DETECTING ENTERING OBJECTS USING THEMETHODS”;

Patent application Ser. No. 09/362,212, which is a Continuation-in-partof U.S. Ser. No. 09/078,521, filed on May 14, 1998, in the names ofWataru Ito, Hirotada Ueda and Hiromasa Yamada and entitled “METHOD OFDISTINGUISHING A MOVING OBJECT AND APPARATUS OF TRACKING AND MONITORINGA MOVING OBJECT”;

Patent application Ser. No. 09/671,178, filed on Sep. 28, 2000, in thenames of Wataru Ito and Hirotada Ueda and entitled “INTRUSION OBJECTDETECTING METHOD AND INTRUSION OBJECT DETECTING APPARATUS”; and

Patent application Ser. No. 09/933,164, filed on August, 2001, in thenames of Wataru Ito and Hirotada Ueda and Toshimichi Okada and entitled“OBJECT DETECTING METHOD AND OBJECT DETECTING APPARATUS AND INTRUDINGOBJECT MONITORING APPARATUS EMPLOYING THE OBJECT DETECTING METHOD”.

BACKGROUND OF THE INVENTION

The present invention relates to a monitoring apparatus using an imagepickup device and particularly to an intruding object detecting methodand an intruding object monitoring apparatus for automatically detectingan object intruding into a monitoring visual field, as a target objectto be detected, from video signals supplied from an image pickup deviceunder a monitoring environment in which the trembling of trees, waves orthe like is also observed.

An intruding object monitoring apparatus using an image pickup devicesuch as a camera as an image input means is to detect an objectintruding into a monitoring visual field or to confirm the kind of theobject to thereby automatically issue a predetermined announcement oralarm without depending on manned monitoring by a watcher which ishetherto done. In order to achieve such a system, there is a method inwhich: an input image obtained from the image input means such as acamera is first compared with a reference background image (that is, animage in which an object to be detected is not picked up) or withanother input image which was obtained at a time different from the timewhen the first-mentioned input image is obtained; a difference betweenthe input image and the reference background image or between the twoinput images is detected for each pixel; and a region having a largedifference is extracted as an object. This method is known as“subtraction method” and has been widely used conventionally.Particularly, the method using the difference between the input imageand the reference background image is known as “background subtractionmethod” and the method using the difference between the input imagesobtained at different times is known as “frame subtraction method”.

The processing by the background subtraction method will be firstdescribed with reference to FIG. 5. FIG. 5 is a diagram for explainingthe principle of processing the object detection according to thebackground subtraction method. In FIG. 5, a reference numeral 101designates an input image; 105, a reference background image; 501, adifference image according to the background subtraction method; 502, abinarized image of the difference image 501; 112, a subtractor; and 115,a binarizer.

In FIG. 5, the subtractor 112 calculates the difference in luminancevalue between two frame images (that is, the input image 101 and thereference background image 105 in FIG. 5) for each pixel to therebyoutput the difference image 501. The binarizer 115 produces thebinarized image 502 in the condition that the pixel value of each pixelof the difference image 501 is set to “0” when it is smaller than apredetermined threshold value Th and the pixel value is set to “255”when it is equal to or greater than the threshold value Th (the pixelvalue of one pixel is calculated on the assumption that each pixel iscomposed of 8 bits).

The human-like object 503 picked up in the input image 101 in thismanner is calculated as a region 504 where a difference is generated bythe subtractor 112. The region 504 is then detected by the binarizer 115as an image 505 indicating a cluster of pixels with the pixel value of“255”. For example, JP-A-9-288732 discloses an application example ofthe background subtraction method.

Next, the processing by the frame subtraction method will be describedwith reference to FIG. 6. FIG. 6 is a diagram for explaining theprinciple of processing the object detection according to the framesubtraction method. In FIG. 6, a reference numeral 101 designates afirst input image; 102, a second input image which is obtained byimaging the same range of visual field as the first input image at atime different from the time when the first input image 101 is obtained;601, a difference image according to the frame subtraction method; 602,a binarized image of the difference image 601; 112, a subtractor; and115, a binarizer.

In FIG. 6, the subtractor 112 calculates the difference in luminancevalue between two frame images (that is, the first input image 101 andthe second input image 102 in FIG. 6) for each pixel and outputs thedifference image 601 in the same manner as that in FIG. 5. The binarizer115 produces the binarized image 602 in the condition that the pixelvalue of each pixel of the difference image 601 is set to “0” when it issmaller than a predetermined threshold value Th and the pixel value isset to “255” when it is equal to or greater than the threshold value Th(the pixel value of one pixel is calculated on the assumption that eachpixel is composed of 8 bits) in the same manner as that in FIG. 5.

The human-like objects 603 and 604 picked up in the first and secondinput images 101 and 102 respectively in this manner are calculated as aregion 605 where a difference is generated by the subtractor 112. Theregion 605 is detected by the binarizer 115 as an image 606 indicating acluster of pixels with the pixel value of “255”. For example,JP-B-2633694 discloses an application example of the frame subtractionmethod.

SUMMARY OF THE INVENTION

The background subtraction method has a feature in that a target objectcan be detected even in the case where the apparent moving velocity ofthe target object on input images is slow. The background subtractionmethod, however, has a problem that a moving object such as trembling ofleaves, waves or the like is detected by mistake if there is such movingobject on the input images. On the other hand, the frame subtractionmethod has a feature in that erroneous detection of moving objects canbe reduced when a time interval for acquiring two frame images to besubjected to a subtraction process is set appropriately (when setting ismade such that the change in trembling of leaves, waves, or the like,between the two frame images becomes small) in the case where there is amoving object such as the trembling of leaves, waves or the like. Theframe subtraction method, however, has a problem that a target objectcannot be detected in the case where the apparent moving velocity of thetarget object to be detected on input images is slow.

An object of the present invention is to provide an intruding objectdetecting method and an intruding object monitoring apparatus fordetecting a target object intruding into an image pickup region whilereducing erroneous detection of moving objects other than the targetobject.

According to an aspect of the present invention, there is provided anintruding object detecting method comprising the steps of: inputtingimages of a monitoring visual field from an image pickup device; storingthe images from the image pickup device in a memory device; calculatingfor each pixel a difference in luminance value between a current inputimage from the image pickup device and each of different input images ina predetermined number of frames greater than one to thereby generaterespective differential images; adding the respective differentialimages, each of which is given weight with predetermined proportion tothereby generate a synthesized differential image; binarizing thesynthesized differential image on the basis of a predetermined thresholdvalue to thereby generate a binarized image; and detecting an object inthe binarized image as an object intruding within the monitoring visualfield.

According to a preferred feature of the present invention, one frame inthe different images in the predetermined number of frames greater thanone is used as a reference background image and the other framesare-used as input images obtained at respective times different from thecurrent time when the current input image is obtained.

The merits and demerits of the frame subtraction method and of thebackground subtraction method are rearranged as follows.

Frame Subtraction Method

Merit: It is possible to reduce an erroneous detection of moving objectsby appropriately setting the time intervals at which images in twoframes used for the subtraction processing are acquired.

Demerit: It is impossible to detect an object making apparently smallmotions (small in the quantity of movement on the image screen at a timeinterval Δt).

Background Subtraction Method

Merit: It is possible to detect even an object making apparently smallmotions (it is also possible to detect an object which stands still).

Demerit: Moving objects other than the target object to be detected maybe erroneously detected.

The inventors of this application have made experiments (frame timeinterval Δt=100 ms) with the frame subtraction method and the backgroundsubtraction method applied to a surveillance ship for detecting anobject intruding a region on the sea. As a result, the followingknowledge has been found.

-   -   In the frame subtraction method, it is possible to suppress        reflection of the setting sun in the surface of the sea (the        area of an error detection region is small even in the case        where the error detection region is detected).    -   In the background subtraction method, it is impossible to        suppress error detection due to reflection of the setting sun        (the area of the error detection region is large).    -   Erroneous detection due to reflection of the setting sun occurs        frequently on this side i.e. foreground side of an image        (because waves look larger as the position on the image becomes        nearer to this side.    -   In the frame subtraction method, it is impossible to detect a        ship at a long distance (because the apparent quantity of        movement of the ship is too small).

The following conclusion has been obtained from these results.

-   -   The frame subtraction method is effective in detecting this side        or foreground of a scene (that is, in detecting a nearer        object).    -   The background subtraction method is effective in detecting the        far side or background of a scene (that is, in detecting a        remoter object).

Therefore, according to a feature of the present invention, the framesubtraction method and the background subtraction method are hybridizedso that the frame subtraction method is used in an image picked up onthis side of a scene by a television camera and the backgroundsubtraction method is used in an image picked up on the far side of thescene to thereby improve intruding object detecting performance.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a diagram for explaining the operation of an intruding objectdetecting process according to the present invention;

FIG. 2 is a flow chart showing an intruding object detecting procedureaccording to a first embodiment of the present invention;

FIG. 3 is a flow chart showing an intruding object detecting procedureaccording to a second embodiment of the present invention;

FIG. 4 is a block diagram showing the hardware configuration of anintruding object monitoring apparatus to which the present invention isapplied;

FIG. 5 is a diagram for explaining the principle of an object detectingprocess in a background frame subtraction method in the related art;

FIG. 6 is a diagram for explaining the principle of an object detectingprocess in a conventional frame subtraction method;

FIG. 7 is a diagram showing an example of an input image in the casewhere the present invention is applied to maritime surveillance;

FIG. 8 is a diagram showing a weighting coefficient image in the casewhere the present invention is applied to maritime surveillance;

FIG. 9 is a diagram showing an example of an input image in the casewhere the present invention is applied to outdoor surveillance;

FIG. 10 is a diagram showing a weighting coefficient image in the casewhere the present invention is applied to outdoor surveillance;

FIGS. 11A to 11D are diagrams for explaining in more detail the settingof the weighting coefficient image depicted in FIG. 8;

FIGS. 12A to 12D are diagrams showing an example in which in theweighting coefficient image, weighting coefficients are set by threevalues;

FIG. 13 is a diagram showing an example in which in a weightingcoefficient image, pixel values are set with multivalues; and

FIG. 14 is a diagram showing an example of how a synthesizeddifferential image is made from differential images and weightingcoefficient images.

DESCRIPTION OF THE EMBODIMENTS

Embodiments of the present invention will be described below withreference to the drawings. In all the drawings, like parts arereferenced correspondingly.

FIG. 4 is a block diagram showing the hardware configuration of anintruding object monitoring apparatus to which the present invention isapplied. First, referring to FIG. 4, the intruding object monitoringapparatus will be described.

In FIG. 4, the intruding object monitoring apparatus has a televisioncamera (hereinafter referred to as TV camera) 401, an image inputinterface 402, a CPU 403, a program memory 404, an image memory 405, awork memory 406, an output interface 407, an image output interface 408,an alarm lamp 409, a monitor 410, and a data bus 411.

The TV camera 401 is connected to the image input interface 402. Themonitor 410 is connected to the image output interface 408. The alarmlamp 409 is connected to the output interface 407. The image inputinterface 402, the CPU 403, the program memory 404, the image memory405, the work memory 406, the output interface 407 and the image outputinterface 408 are connected to the data bus 411.

In FIG. 4, the TV camera 401 picks up an image in an image pickup visualfield including a region to be monitored. The TV camera 401 converts thepicked-up image into a video signal and supplies the video signal to theimage input interface 402. The image input interface 402 converts theinput video signal into image data of a format (for example, with awidth of 320 pixels, a height of 240 pixels and a depth of 8 bit/pixel)allowed to be dealt with by the intruding object monitoring apparatusand delivers the image data to the image memory 405 through the data bus411. The image memory 405 stores the image data supplied from the imageinput interface 402.

The CPU 403 analyzes images stored in the image memory 405 by using thework memory 406 in accordance with an operating program retained in theprogram memory 404. As a result of the analysis, the CPU 403 obtainsinformation as to whether an object intrudes into the image pickupvisual field of the TV camera 401 or not. The CPU 403 displays, forexample, a processed result image on the monitor 410 through the imageoutput interface 408 from the data bus 411 and turns the alarm lamp 409on through the output interface 407.

The image output interface 408 converts a signal of the CPU 403 into asignal of a format (for example, NTSC video signal) allowed to be usedby the monitor 410 and delivers the converted signal to the monitor 410.The monitor 410 displays, for example, an intruding object detectingresult image.

FIG. 2 is a flow chart showing an intruding object detecting procedureaccording to a first embodiment of the present invention. This flow isexecuted by use of the hardware configuration of the intruding objectmonitoring apparatus shown in FIG. 4.

The procedure shown in the flow chart of FIG. 2 is an intruding objectdetecting method comprising the steps of: calculating a differentialimage between an input image 101 from the TV camera 401 shown in FIG. 4and each of previous input images in a predetermined number of frames(greater than one) stored in the image memory 405 by a frame subtractionmethod shown in FIG. 6; adding the thus obtained differential images inthe predetermined number of frames while weighting the respectivedifferential images to thereby generate a synthesized differentialimage; binarizing the synthesized differential image on the basis of apredetermined threshold value; and detecting an object intruding intothe visual field of the TV camera 401 on the basis of the binarizedimage.

First, in an image input step 201, an input video signal of an imagepicked up by the TV camera 401 is obtained as an input image 101, forexample, of 320×240 pixels. Then, in a frame counter clearing step 202,the value i of a frame counter, which is a variable used for managingthe number of the image to be subjected to the frame subtraction, is setto “1”.

Then, in a frame subtraction step 203, a difference (hereinafterrepresented by ci(x, y) in which i is the value of the frame counter,and (x, y) indicates the position of the pixel on the image) for eachpixel between the input image 101 (here, represented by a(x, y)) and theprevious input image (here, represented by bi(x, y)) retained in theimage memory 405 is calculated.

At this time, the input image to be subjected to the differencecalculation retained in the image memory 405 is determined on the basisof the frame number. When, for example, the value i of the frame counteris “1”, the input image is an input image b1(x, y) which is the one mostrecently stored in the image memory 405 (i.e. one frame before the inputimage 101). The difference for each pixel is calculated as follows.Ci(x, y)=|a(x, y)−bi(x, y)|  (1)

Then, in the frame counter increment step 204, the value of the framecounter is increment by one.

In the frame termination judging step 205, process goes to the framesubtraction step 203 when the value of the frame counter is smaller thana predetermined value N (for example, N=3), and goes to a differentialimage synthesizing step 206 when the value of the frame counter is equalto or greater than the predetermined value N. Here, the predeterminedvalue N indicates the number of frames to be subjected to the framesubtraction, namely, the number of the input images to be retained inthe image memory 405. For example, when N=4, it means that the number ofthe input images retained in the image memory 405 is 4. In this case,differential images in 4 frames (ci(x, y) in which i is an integer offrom 1 to 4) are obtained.

Then, in the differential image synthesizing step 206, the obtaineddifferential images in N frames are added together while being weightedwith a predetermined weighting coefficient image di(x, y) (which will bedescribed later) to thereby obtain a synthesized differential image e(x,y). The weighting coefficient image is defined in FIG. 14. Thesynthesized differential image e(x, y) is calculated as represented bythe following expression:

$\begin{matrix}{{e\left( {x,y} \right)} = {\frac{1}{255}{\sum\limits_{i = 1}^{N}{{{di}\left( {x,y} \right)}^{*}{{ci}\left( {x,y} \right)}}}}} & (2)\end{matrix}$in which the weighting coefficient image di(x, y) is previously set asfollows.

$\begin{matrix}{{\sum\limits_{i = 1}^{N}{{di}\left( {x,y} \right)}} \leq 255} & (3)\end{matrix}$

The weighting coefficient image di(x, y) indicates the rate ofcontribution by which each differential image ci(x, y) contributes tothe synthesized differential image e(x, y). For example, when d1(100,100)=255, it means that the rate of contribution of the firstdifferential image c1(x, y) to the synthesized differential image e(x,y) is 100% in the coordinates (100, 100). (The weighting coefficientimage is expressed as an image having pixels each composed of 8 bits.When the pixel value of the weighting coefficient image is “0”, it meansthat the rate of contribution is 0%. On the other hand, when the pixelvalue is “255”, it means that the rate of contribution is 100%.)

FIG. 14 shows an example in which the number of frames of thedifferential images is 2, namely, ci(x, y), where i=1, 2. For brevity'ssake, explanation will be made focusing on pixel positions (1)–(4) ofeach of differential images, weighting coefficient image and synthesizedimage.

In FIG. 14, luminance values (pixel values) of the differences atrespective pixel positions (1)–(4) of the background differential imagec1(x, y) are outputted to a multiplier 140 and luminance values (pixelvalues) of the differences at respective pixel positions (1)–(4) of theframe differential image c2(x, y) are outputted to a multiplier 141.Further, in the weighting coefficient image d1(x, y), weightingcoefficients at the same pixel positions (1)–(4) as those of thebackground differential image are given values having the same dimensionas luminance values. For example, the weighting coefficient d1(1) at thepixel position (1) is given “255”, d1(2) at the pixel position (2) isgiven “127”, d1(3) at the pixel position (3) is given “127” and d1(4) atthe pixel position (4) is given “0”. Similarly, in the weightingcoefficient image d2(x, y), d2(1) is given “0”, both of d2(2) and d2(3)are given “128” and d2(4) is given “255”.

Therefore, by carrying out a multiplying operation pixel by pixel withthe multipliers 140 and 142, adding together the outputs of themultipliers with an adder 142 and dividing the output of the adder by“255”, the synthesized differential image e(x, y) is obtained.

Next, the setting of the weighting coefficient image will be furtherdescribed below with reference to FIGS. 7 to 10 and FIGS. 11A to 11D,FIGS. 12A to 12D and FIG. 13. FIGS. 7 and 8 show an example of thesetting of the weighting coefficient image in the case where the presentinvention is applied to maritime surveillance. In FIG. 7, 701 denotes aninput image obtained by imaging the range of the visual field to bemonitored. FIG. 8 shows a scene having weighting coefficient imagesdi(x, y) displayed in superposition in a range of i of from 1 to 4 inthe case of the value N=4. In this example of FIG. 8, the scene isdivided into the region of the surface of the sea and the other region804 consisting of a seawall and a lighthouse. The region of the surfaceof the sea is further divided into three sub-regions 801 to 803 inaccordance with the distance from the TV camera 401.

The trembling of waves occurring on the surface of the sea is observedmore largely as the position goes nearer to the TV camera 401.Therefore, the frame subtraction needs to be done in such a manner thatthe change in the luminance value due to the trembling of waves may bereduced in a zone nearer to the TV camera 401. Hence, the time intervalfor inputting images of two frames to be subjected to the framesubtraction needs to be shortened. That is, the differential images areset so that the differential image c1(x, y) is used (i.e. inputting oftwo-frame images at short interval of e.g. 100 msec) for a zone 801 ofthe surface of the sea on this side of the scene, the differential imagec2(x, y) is used (i.e. inputting of two-frame images at intermediateinterval of e.g. 500 msec) for a zone 802 far (for example, by 30 m ormore) from the TV camera 401, and the differential image c3(x, y) isused (i.e. inputting of two-frame images at long interval of e.g. 3 sec)for a zone 803 farther (for example, by 100 m or more) from the TVcamera 401. For a zone 804 in which there is no trembling of waves,however, the differential image c4(x, y) is used because the timeinterval for inputting images of two frames can be made long.Accordingly, the weighting coefficient image d1(x, y) may be set suchthat the values of pixels in the zone 801 to “255” and the values ofpixels in the zones 802 to 804 to “0”.

Similarly, the weighting coefficient image d2(x, y) may be set such thatthe values of pixels in the zone 802 to “255” and the values of pixelsin the zones 801, 803 and 804 to “0”. The weighting coefficient imaged3(x, y) may be set such that the values of pixels in the zone 803 to“255” and the values of pixels in the zones 801, 802 and 804 to “0” 1.The weighting coefficient image d4(x, y) may be set such that the valuesof pixels in the zone 804 to “255” and the values of pixels in the zones801 to 803 to “0”.

In this manner, the weighting coefficient images d1(x, y) to d4(x, y)are drawn as shown in FIGS. 11A to 11D respectively. FIGS. 11A to 11Dshow an example in which the values of pixels in the weightingcoefficient images di(x, y) are set by two values “0” and “255” in thescene shown in FIG. 7. In FIG. 11A, the image 1101 expresses theweighting coefficient image d1(x, y), which sets pixel values in zones1101 a and 1101 b to “255” and pixel values in the remaining zone to“0”. In FIG. 11B, the image 1102 expresses the weighting coefficientimage d2(x, y), which sets pixel values in zones 1102 a and 1102 b to“255” and pixel values in the remaining zone to “0”. In FIG. 11C, theimage 1103 expresses the weighting coefficient image d3(x, y), whichsets pixel values in a zone 1103 a to “255” and pixel values in theremaining zone to “0”. In FIG. 11D, the image 1104 expresses theweighting coefficient image d4(x, y), which sets pixel values in a zone1104 a to “255” and pixel values in the remaining zone to “0”.

It is a matter of course that the values of pixels near to the boundarybetween zones may be set to be smaller than “255”. For example, d1(x,y)=128 and d2(x, y)=127 may be applied to pixels corresponding to theboundary between the zones 801 and 802. That is, the weightingcoefficient images may be drawn as shown in FIGS. 12A to 12Drespectively.

FIGS. 12A to 12D show an example in which the width of the boundary isset to 30 pixels and in which values of the pixels in the weightingcoefficient images di(x, y) are set by three values “0”, “127” and “255”in the scene shown in FIG. 7. (Because the maximum pixel value “255”cannot be divided by “2”, the remainder generated by the distribution ofthe weighting coefficients (contribution rates) is allocated to any oneof the weighting coefficient images. Hence, the difference between thepixel values “127” and “128” in the weighting coefficient images is only0.4% with respect to the maximum weighting coefficient “255”, so thatthe pixel values “127” and “128” can be regarded as one weightingcoefficient. Therefore, the pixel value “127” is used in this case.) Theimage 1201 expresses the weighting coefficient image d1(x, y), whichsets pixel values in zones 1201 a and 1201 b to “255”, pixel values inzones 1201 c and 1201 d to “127” and pixel values in the remaining zoneto “0”. The image 1202 expresses the weighting coefficient image d2(x,y), which sets pixel values in zones 1202 a and 1202 b (the same as thezones 1201 c and 1201 d respectively) to “128”, pixel values in zones1202 c and 1202 d to “255”, pixel values in zones 1202 e and 1202 f as“127” and pixel values in the remaining zone to “0”. The image 1203expresses the weighting coefficient image d3(x, y), which sets pixelvalues in zones 1203 a and 1203 b (the same as the zones 1202 e and 1202f respectively) to “128”, pixel values in a zone 1203 c to “255” andpixel values in the remaining zone to “0”. The image 1204 expresses theweighting coefficient image d4(x, y), which sets pixel values in zones1204 a and 1204 b to “255” and pixel values in the remaining zone to“0”. Note that in these setting examples, d4(x, y) is expressed by twovalues, namely, “0” and “255”because the region 804 consisting of thebreakwater and lighthouse does not have the characteristic that thelower a position in the image becomes, i.e. the shorter the distancefrom the camera becomes, the larger the wave appears as is the case withthe other regions 801–803 and it may be sufficient that a single frametime-interval (i.e. frame subtraction) or a background subtraction isapplied.

Although FIGS. 11A to 11D and FIGS. 12A to 12D show the case where theweighting coefficients of each weighting coefficient image are set bytwo or three values, any other weighting coefficient setting method maybe used. An example of the weighting coefficient setting method will bedescribed with reference to FIG. 13. FIG. 13 shows an example in whichpixel values in each weighting coefficient image are allocated to 256values in a range of from 0 to 255. In FIG. 13, the image 1301 expressesthe same scene as that in FIG. 7, and the graph 1302 expresses thedistribution of contribution rates. In the graph 1302 expressing thedistribution of contribution rates, the vertical position corresponds tothe y ordinate of the image 1301 and the horizontal width expresses therate of contribution (the value of the weighting coefficient) to thesynthesized differential image e(x, y). The graph 1302 is divided intothree zones 1302 a, 1302 b and 1302 c, which correspond to the weightingcoefficient images d1(x, y), d2(x, y) and d3(x, y) of the differentialimages c1(x, y), c2(x, y) and c3(x, y) respectively. The zones 1302 aand 1302 b are separated from each other by a line connecting a point1302 g (corresponding to the y ordinate 220) and a point 1302 h(corresponding to the y ordinate 80). The zones 1302 b and 1302 c areseparated from each other by a line connecting a point 1302 i(corresponding to the y ordinate 120) and a point 1302 j (correspondingto the y ordinate 20). These points 1302 g to 1302 j are setexperimentally in accordance with the distance from the TV camera 401.For example, the point 1302 g is set so as to correspond to the yordinate on the image in accordance with the distance of 10 m from theTV camera 401. Similarly, the points 1302 i, 1302 h and 1302 j are setrespectively so as to correspond to the y ordinate on the image inaccordance with the distance of 30 m from the TV camera 401, thedistance of 80 m from the TV camera 401, and the distance of 150 m fromthe TV camera 401. When the image is divided into zones as shown in FIG.13, the widths of the zones 1302 a, 1302 b and 1302 c (that is, theweighting coefficients of d1(x, y), d2(x, y) and d3(x, y)) can becalculated as follows.

$\begin{matrix}{{d_{1}\left( {x,y} \right)} = \left\{ \begin{matrix}0 & \left( {0 \leq y < 80} \right) \\{255\frac{y - 80}{220 - 80}} & \left( {80 \leq y < 220} \right) \\255 & \left( {220 \leq y \leq 255} \right)\end{matrix} \right.} & (4) \\{{d_{1}\left( {x,y} \right)} = \left\{ \begin{matrix}0 & \left( {0 \leq y < 20} \right) \\{255\frac{y - 20}{120 - 20}} & \left( {20 \leq y < 80} \right) \\{255\left( {\frac{y - 20}{120 - 20} - \frac{y - 80}{220 - 80}} \right)} & \left( {80 \leq y < 120} \right) \\{255\frac{220 - y}{220 - 80}} & \left( {120 \leq y < 220} \right) \\0 & \left( {220 \leq y \leq 255} \right)\end{matrix} \right.} & (5) \\{{d_{1}\left( {x,y} \right)} = \left\{ \begin{matrix}255 & \left( {0 \leq y < 20} \right) \\{255\frac{y - 20}{120 - 20}} & \left( {20 \leq y < 120} \right) \\255 & \left( {120 \leq y \leq 255} \right)\end{matrix} \right.} & (6)\end{matrix}$

Here, when, for example, weighting coefficients in the position 1301 a(y=100) of the image 1301 are calculated, d1(x, y)=36 (width 1302 d),d2(x, y)=168 (width 1302 e) and d3(x, y)=51 (width 1302 f) are obtained.Incidentally, weighting coefficients in the zone 804 in which there isno trembling of waves (that is, to which the background subtractionmethod can be applied) are set as di(x, y)=0 (i<4) and d4(x, y)=255.Although this embodiment has shown the case where the zones 1302 a, 1302b and 1302 c determining the contribution rates of the weightingcoefficient images are separated from one another by lines connectingthe reference points 1302 g, 1302 h, 1302 i and 1302 j as shown in thegraph 1302, the present invention may be applied also to the case wherethe zones are separated from one another by curves.

FIGS. 9 and 10 show an example of the setting of weighting coefficientimages in the case where the present invention is applied to outdoorsurveillance. FIG. 9 shows an input image 901. FIG. 10 shows an exampleof N=3, that is, the case where weighting coefficient images di(x, y),i=1 to 4, are displayed in superposition. In this example, the image isdivided into a building/land/sky zone and a tree/plant zone. Thetree/plant zone is further divided into two parts by kind of tree andplant.

In the example shown in FIG. 9, the apparent magnitude of motion on theimage is set so that the motion of trees on the upward portion of theimage is larger than the motion of plants on the center portion of theimage. In the zone in which trembling is large, the time interval forinputting images of two frames to be subjected to the frame subtractionneeds to be shortened to reduce the change of the trembling of trees.That is, setting is made so that the differential image c1(x, y) is usedfor the tree zone 1002 and the differential image c2(x, y) is used forthe plant zone 1001. For the zone 1003 in which there is no trembling oftrees, however, the differential image c3(x, y) is used because the timeinterval for inputting images of two frames can be made long. Hence, theweighting coefficient image d1(x, y) sets pixel values in the zone 1002to “255” and pixel values in the zones 1001 and 1003 to “0”.

Similarly, the weighting coefficient image d2(x, y) sets pixel values inthe zone 1001 to “255” and pixel values in the zones 1002 and 1003 to“0”. The weighting coefficient image d3(x, y) sets pixel values in thezone 1003 to “255” and pixel values in the zones 1001 and 1002 to “0”.It is a matter of course that a weighting coefficient smaller than “255”may be set for pixels near the boundary between adjacent ones of thezones in the same manner as in FIGS. 7 and 8. For example, c1(x, y)=128and c2(x, y)=127 may be set for pixels corresponding to the boundarybetween the zones 1001 and 1002.

Furthermore, as shown in FIG. 13, 256 values in a range of from 0 to 255may be allocated to the weighting coefficient images. Although FIG. 13shows the case where weighting coefficients are allocated in accordancewith the distance from the camera 401, FIG. 9 shows the case whereweighting coefficients are allocated in accordance with the degree ofmotion of an object observed on the image picked up by the TV camera401. (Setting is made so that the contribution rate of d1(x, y) short inthe frame interval used in the subtraction method becomes high in thezone (for example, zone 1002) where the object making large motions isobserved, whereas the contribution rate of d3(x, y) long in the frameinterval used in the subtraction method or as a difference between theinput image and the reference background image becomes high in the zone(for example, zone 1003) where the object making little motions isobserved.)

Note that it may be sufficient that the weighting coefficient image isset once when installing the intruding object monitoring apparatus. Forthis reason, the step of setting the weighting coefficient image is notshown in the flow chart of FIG. 2 as well as in the flow chart of FIG. 3to be described later.

Then, in a binarizing step 207 in FIG. 2, the synthesized differentialimage e(x, y) obtained by the differential image synthesizing step 206is binarized by use of a predetermined threshold value Th (for example,Th=20) so that the pixel value for each pixel of the synthesizeddifferential image e(x, y) (the pixel value for each pixel is calculatedon the assumption that each pixel is composed of 8 bits) is set to “0”when the pixel value is smaller than the threshold value Th and as “255”when the pixel value is equal to the threshold value Th or greater.Thus, a binarized image f(x, y) is obtained.

Then, in an intruding object judging step 208, a judgment is made as towhether a cluster of pixels each having the pixel value “255” is presentin the thus obtained binarized image f(x, y) or not (that is, whether acluster of pixels equal to or greater than a predetermined number ofpixels (for example, 100 pixels) is present or not). When a cluster ofpixels each having the pixel value “255” is present, the cluster isregarded as an intruding object and process goes to an alarm/monitordisplay step 210 from the branch step 209. When there is no cluster ofpixels each having the pixel value “255”, process goes to the inputimage saving step 211.

In an alarm/monitor display step 210, the alarm lamp 409 is turned onthrough the output interface 407 or, for example, a monitoring result isdisplayed on the monitor 410 through the image output interface 408.

Then, in an input image saving step 211, the input image 101 is retainedin the image memory 405 as an one frame earlier input image b1(x, y). Atthis time, input images b1(x, y) to bN−1(x, y) which have beenpreviously retained are copied as input images b2(x, y) to bN(x, y)respectively. In this manner, input images up to a N frame earlier inputimage can be retained in the image memory 405. Note that in the inputimage saving step 211 the input image 101 may be retained in the imagememory 405 one frame by one frame or at intervals of 100 msec. Further,the input image saving step 211 may be placed before the differentialimage synthesizing step 206 in which case however input images arestored twice, namely, in the image memory 405 in the image input step201 and again stored in the input image saving step 211, to therebywastefully use the image memory 405.

In such a manner, any other moving object than the target object in thevisual field of the image pickup device can be prevented from appearingas a difference in a differential image, so that accurate intrudingobject detection can be made.

FIG. 3 is a flow chart showing an intruding object detecting procedureaccording to a second embodiment of the present invention. FIG. 3 isobtained by adding a background subtraction step 301 and a referencebackground image updating step 302 to the flow chart shown in FIG. 2.

In the background subtraction step 301, a difference for each pixelbetween the input image 101 and the reference background image 105 iscalculated as c(x,y). In the differential image synthesizing step 206,the differential image c(x, y) obtained by the background subtraction issynthesized instead of using the differential image between the currentinput image and an input image of the N-th frame as explained above inthe flow chart of FIG. 2. At this time, the background differentialimage c(x, y) obtained by background subtraction is applied to the zone804 of FIG. 8 in the flow chart of FIG. 3 though the differential imagec4(x, y) 4 frames before was applied to the zone 804 of FIG. 8 in theflow chart of FIG. 2.

In the reference background image updating step 302, for example, pixelsof the input image and pixels of the reference background image areaveraged to generate a new reference background image. Because the othersteps in the flow chart of FIG. 3 are the same as those in the flowchart of FIG. 2, description thereof will be omitted.

This series of processing flows will be described below with referenceto FIG. 1. FIG. 1 shows an example in which three frames are used forthe frame subtraction and the background subtraction is also used(namely, N=4). In FIG. 1, the image 101 represents a current inputimage, the image 102 represents an image inputted at a time differentfrom that at which the input image 101 was inputted (for example, aninput image inputted one frame earlier), the image 103 represents animage inputted at a time further different from that at which the inputimage 101 was inputted (for example, an input image inputted two framesearlier), the image 104 represents an image inputted at a time stillfurther different from that at which the input image 101 was inputted(for example, an input image inputted three frames earlier), and theimage 105 represents a reference background image. Further, the image106 represents a weighting coefficient image for a differential imagebetween the current input image 101 and the input image 102, the image107 represents a weighting coefficient image for a differential imagebetween the current input image 101 and the input image 103, the image108 represents a weighting coefficient image for a differential imagebetween the current input image 101 and the input image 104, and theimage 109 represents a weighting coefficient image for a differentialimage between the current input image 101 and the reference backgroundimage 105.

A difference for each pixel between the current input image 101 and theinput image 102 is calculated by a subtractor 112-1. The product of thethus obtained differential image and the weighting coefficient image 106for each pixel is calculated by a multiplier 113-1 and supplied to anadder 114. A difference for each pixel between the current input image101 and the input image 103 is calculated by a subtractor 112-2. Theproduct of the thus obtained differential image and the weightingcoefficient image 107 for each pixel is calculated by a multiplier 113-2and supplied to the adder 114. A difference for each pixel between thecurrent input image 101 and the input image 104 is calculated by asubtractor 112-3. The product of the thus obtained differential imageand the weighting coefficient image 108 for each pixel is calculated bya multiplier 113-3 and supplied to the adder 114. A difference for eachpixel between the current input image 101 and the background image 105is calculated by a subtractor 112-4. The product of the thus obtaineddifferential image and the weighting coefficient image 109 for eachpixel is calculated by a multiplier 113-4 and supplied to the adder 114.

In the adder 114, the supplied differential images of 4 frames are addedtogether for each pixel to thereby obtain a synthesized differentialimage 110. Each pixel in the synthesized differential image 110 thusobtained is compared with a predetermined threshold value by thebinarizer 115. If the pixel value of the pixel is equal to or greaterthan the threshold value, it is set to “255”. On the other hand, if thepixel value is less than the threshold value, it is set to “0”. Thus, abinarized image 111 is obtained. In such a manner, any other movingobject than the target object existing in the visual field of the imagepickup device can be prevented from appearing as a difference in adifferential image, so that accurate intruding object detection can bemade.

Hence, in accordance with the embodiments of the present invention,frame subtraction images obtained from input images at different frametime intervals and a background subtraction image between the inputimage and the reference background image are synthesized by usingpredetermined weighting coefficients respectively. Hence, any movingobjects such as leaves or waves other than the target object in themonitoring visual field to be monitored can be prevented from appearingas a difference in a differential image, so that the range ofapplication of the intruding object detecting apparatus can be widened.

According to the present invention, there can be provided an intrudingobject detecting method and an intruding object monitoring apparatus fordetecting a target object intruding into an image pickup region whilereducing the error detection of moving objects other than the targetobject.

1. An object detecting method for detecting an object in an imageobtained from an image pickup means, comprising: a frame subtractionstep of executing a plurality of frame subtraction processings eachframe subtraction processing being between an input image from the imagepickup means and respective ones of a plurality of images each having adifferent time interval with respect to the time interval of said inputimage; a synthesizing step of adding together a plurality ofdifferential images obtained by said frame subtraction processings basedon coefficients which are set for respective ones of predeterminedregions of the image; and an object detecting step of detecting anobject based on data obtained from said synthesizing step.
 2. An objectdetecting method according to claim 1, wherein said coefficients are setbased on a distance from the image pickup means.
 3. An object detectingmethod according to claim 1, wherein said coefficients are set based ona magnitude of movement of an object in a respective one ofpredetermined regions of said image.
 4. An object detecting method fordetecting an object in an image obtained from an image pickup means,comprising: a frame subtraction step of executing a plurality of framesubtraction processings each frame subtraction processing being for eachof a plurality of predetermined regions, wherein each of thepredetermined regions has a frame time interval which is changed fromthe frame time intervals of each of the other predetermined regions; andan object detecting step of detecting an object based on a plurality ofdifferential images obtained from said frame subtraction proessings. 5.An object detecting method according to claim 4, wherein said frame timeinterval is set based on a distance from said image pickup means.
 6. Anobject detecting method according to claim 4, wherein said frame timeinterval is set based on a magnitude of movement of an object tinrespective one of predetermined regions of said image.
 7. An objectdetecting apparatus for detecting an object in an image obtained fromimage pickup means, comprising: frame subtraction means for executing aplurality of frame subtraction processings each frame subtractionprocessing being between an input image from the image pickup means andrespective ones of a plurality of images each having a different timeinterval with respect to the time interval of said input image;synthesizing means for adding together a plurality of differentialimages obtained by said frame subtraction processings based oncoefficients which are set for respective ones of predetermined regionsof the image; and object detecting means for detecting an object basedon data obtained from said synthesizing means.
 8. An object detectingapparatus according to claim 7, wherein said coefficients are set basedon a distance from said image pickup means.
 9. An object detectingapparatus according to claim 7, wherein said coefficients are set basedon a magnitude of movement of an object in respective one ofpredetermined regions of said image.
 10. An object detecting apparatusfor detecting an object in an image obtained from an image pickup means,comprising: frame subtraction means for executing a plurality of framesubtraction processings each frame subtraction processing being for eachof a plurality of predetermined regions, wherein each of thepredetermined regions has a frame time interval which is changed fromthe frame time intervals of each of the other predetermined regions; andobject detecting means for detecting an object based on a plurality ofdifferential images obtained from said frame subtraction processingsperformed by said frame subtraction means.
 11. An object detectingapparatus according to claim 10, wherein said frame time interval is setbased on a distance from said image pickup means.
 12. An objectdetecting apparatus according to claim 10, wherein said frame timeinterval is set based on a magnitude of movement of an object in arespective one of the predetermined regions of said image.